Image processing device and image processing program

ABSTRACT

An image processing unit of the present invention receives a first image, performs weighted addition on color information in the first image to generate a color component different from that of the color information in the first image, and outputs the generated color component as a second image. The first image is expressed in a plurality of color components and comprises a plurality of pixels each having color information corresponding to one of the color components. At least nine coefficient patterns consisting of values of not less than zero are prepared, and any one of the coefficient patterns is used for weighted addition. This allows an improvement in image quality with high precision.

TECHNICAL FIELD

The present invention relates to an image processing apparatus forenhancing an image that is expressed in a plurality of color componentsand comprises a plurality of pixels each having color informationcorresponding to one of the color components. The present invention alsorelates to an image processing program for making a computer enhance animage that is expressed in a plurality of color components and consistsof a plurality of pixels each having color information corresponding toone of the color components.

BACKGROUND ART

Some electronic cameras generate color image data by using an imagesensor on which color filters in three colors (R, G, B: red, green,blue) are arranged on predetermined positions (such as in a Bayerarray).

In this kind of electronic cameras, each of the pixels of the imagesensor outputs color information on a single color component alone.Thus, processes for enhancing the image by providing each pixel withcolor information corresponding to three color components have beenpracticed.

Among such processes heretofore practiced is a color interpolationprocess. Specifically, similarity of each pixel in conjunction with thedirections to its adjacent pixels is judged, and interpolation values(values that are equivalent to the color information corresponding tocolor components not contained in respective pixels) are obtained inaccordance with the judgement result.

Moreover, Japanese Unexamined Patent Application Publication No.2000-78597 (U.S. Pat. No. 6,075,889) discloses technology for performinga process of generating a luminance value and a “chrominance value” foreach pixel. That is, enhancement of an image is achieved through theprocessing in which each pixel is provided with the color components ofa colorimetric system different from the colorimetric system that wasadopted in the image at the time of generation by the image sensor.Here, the luminance values of the respective pixels are generated, beingclassified into five similarity directions, which are a horizontaldirection, vertical direction, flat, diagonal direction 1, and diagonaldirection 2.

According to the technology disclosed in the foregoing publication,however, luminance values are basically generated from color informationcorresponding to two color components, as is evident from how theluminance values of pixels containing color information corresponding tothe red component or color information corresponding to the bluecomponent are generated. Consequently, the luminance values generated atthe respective pixels become uneven in RGB ratios pixel by pixel, andthe RGB ratios of the generated luminance values vary greatly each timethe target pixel of the luminance value generation moves on. Thus, therehas been a high possibility that false structures not existing in thephotographic object (false structures attributable to the Bayer array)appear as structures varying pixel by pixel over chromatic areas andflat areas in particular.

Moreover, according to the technology disclosed in the foregoingpublication, the generated luminance values are corrected withcorrection values which vary depending on the directions of similarityat the respective pixels. Such correction values are extracted from“blurred luminance values” whose high frequency components pertaining tostructural factors are all broken. Hence, the high frequency componentshave been corrected only insufficiently, with the results lacking inhigh frequency components.

Furthermore, according to the technology disclosed in the foregoingpublication, since chrominance is generated with reference to luminancevalues that are generated from color information over a wide range,local high frequency components tend to disappear and color artifactseasily occur.

That is, it has been impossible to expect image enhancement with highprecision from the technology disclosed in the foregoing publication.

DISCLOSURE OF THE INVENTION

It is an object of the present invention to provide an image processingapparatus which can achieve a high-precision enhancement of an imagethat is expressed in a plurality of color components and consists of aplurality of pixels each having color information corresponding to oneof the color components.

Another object of the present invention is to provide an imageprocessing program which can achieve, by using a computer, ahigh-precision enhancement of an image that is expressed in a pluralityof color components and consists of a plurality of pixels each havingcolor information corresponding to one of the color components.

Hereinafter, the subject matter of the present invention will bedescribed below.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and which to use from the at least nine coefficientpatterns is selected in accordance with the determined result.

According to another aspect of the present invention, in the imageprocessing unit, at least nine coefficient patterns are prepared forperforming weighted addition on color information present at a targetpixel in the first image and color information present at pixelsadjoining the target pixel.

According to another aspect of the present invention, in the imageprocessing unit, when the first image is expressed in a first colorcomponent set with a higher pixel density and second and third colorcomponents set with a lower pixel density, weighted addition on a pixelhaving the first color component is performed by using a coefficientpattern prepared separately from the at least nine coefficient patterns.

-   (4) According to another image processing apparatus of the present    invention as set forth in (1), in the image processing unit, when    the first image is expressed in a first color component set with a    higher pixel density and second and third color components set with    a lower pixel density, weighted addition on a pixel having the first    color component is performed by using a coefficient pattern prepared    separately from the at least nine coefficient patterns.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, for a pixel having the second or third color componentand adjacent to pixels having the first color component. When the levelsof similarity are indistinguishable along any direction, a coefficientpattern including weighted addition on color information present at aplurality of pixels having the first color component is used as thecoefficient pattern prepared separately.

According to another aspect of the present invention, in the imageprocessing unit, at the time of the weighted addition, a coefficientpattern including weighted addition on color information present at atarget pixel and color information present at the pixels having thefirst color component and lying the closest to the target pixel is used,as the coefficient pattern including the weighted addition on the colorinformation present at the plurality of pixels having the first colorcomponent.

-   (7) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, performing weighted addition on color information in the    first image by using variable coefficients of not less than zero to    generate a color component different from that of the color    information in the first image, and outputting the generated color    component as a second image. The first image is expressed in a    plurality of color components and comprises a plurality of pixels    each having color information corresponding to one of the color    components. In the image processing unit, weighted addition on color    information present at a target pixel in the first image and color    information present at pixels adjoining the target pixel is    performed.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the coefficients of the weighted addition arechanged in accordance with the determined result.

-   (9) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, performing weighted addition on color information in the    first image by using variable coefficients of not less than zero to    generate a color component different from that of the color    information in the first image, and outputting the generated color    component as a second image. The first image is expressed in three    or more types of color components and comprises a plurality of    pixels each having color information corresponding to one of the    color components. In the image processing unit, weighted addition on    color information corresponding to at least three types of color    components in the first image is performed over all the pixels in    the first image.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the coefficients of the weighted addition arechanged in accordance with the determined result.

According to another aspect of the present invention as set forth in(9), in the image processing unit, weighted addition on colorinformation present at pixels in a narrowest range around a target pixelin the first image is performed. The narrowest range includes the colorinformation corresponding to the at least three types of colorcomponents.

-   (12) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, performing weighted addition on color information in the    first image by using variable coefficients of not less than zero to    generate a color component different from that of the color    information in the first image, and outputting the generated color    component as a second image. The first image is expressed in a    plurality of color components and comprises a plurality of pixels    each having color information corresponding to one of the color    components. In the image processing unit, weighted addition on the    color information in the first image in constant color-component    ratios is performed over all the pixels in the first image.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the coefficients of the weighted addition arechanged in accordance with the determined result.

According to another aspect of the present apparatus of the presentinvention, in the image processing unit, when the first image isexpressed in a first color component set with a higher pixel density andsecond and third color components set with a lower pixel density,weighted addition on color information corresponding to the second colorcomponent and color information corresponding to the third colorcomponent is performed at an identical color-component ratio.

-   (15) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, performing weighted addition on color information in the    first image by using variable coefficients of not less than zero to    generate a color component different from that of the color    information in the first image, performing filter processing with    predetermined fixed filter coefficients to correct the color    component different from that of the color information in the first    image, and outputting the generated color component as a second    image. The first image is expressed in a plurality of color    components and comprises a plurality of pixels each having color    information corresponding to one of the color components.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the coefficients of the weighted addition arechanged in accordance with the determined result.

According to another aspect of the present invention, in the imageprocessing unit, filter coefficients including positive and negativevalues are used as the predetermined fixed filter coefficients.

-   (18) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, generating a luminance component different from color    information in the first image at the same pixel positions as in the    first image by using the color information in the first image,    generating a chrominance component different from the color    information in the first image at the same pixel positions as in the    first image separately from the luminance component, and outputting    the luminance component and the chrominance component as a second    image. The first image is expressed in a plurality of color    components and comprises a plurality of pixels each having color    information corresponding to one of the color components. In the    image processing unit, weighted addition on color information in the    first image is performed by using variable coefficients of not less    than zero to generate the luminance component.-   (19) Another image processing apparatus according to the present    invention includes an image processing unit for receiving a first    image, generating a luminance component different from color    information in the first image at all of the same pixel positions as    in the first image by using the color information in the first    image, generating a chrominance component different from the color    information in the first image at the same pixel positions as in the    first image separately from the luminance component, and outputting    the luminance component and the chrominance component as a second    image. The first image is expressed in a plurality of color    components and comprises a plurality of pixels each having color    information corresponding to one of the color components.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the luminance component is generated by performingweighted addition on the color information in the first image inaccordance with the determined result.

According to another aspect of the present invention, in the imageprocessing unit, levels of similarity along a plurality of directionsare determined, and the chrominance component is generated by performingweighted addition on the color information in the first image inaccordance with the determined result.

According to another aspect of the present invention, in the imageprocessing unit, when the first image is expressed in a first colorcomponent, a second color component, and a third color component,similarity factors along a plurality of directions are calculated byusing at least one similarity factor component out of:

a first similarity factor component consisting of the first colorcomponent and the second color component;

a second similarity factor component consisting of the second colorcomponent and the third color component;

a third similarity factor component consisting of the third colorcomponent and the first color component;

a fourth similarity factor component consisting of the first colorcomponent alone;

a fifth similarity factor component consisting of the second colorcomponent alone; and

a sixth similarity factor component consisting of the third colorcomponent alone, and the levels of similarity along the plurality ofdirections are determined based on the similarity factors.

According to another aspect of the present invention, in the imageprocessing unit, the color component of the second image is outputted inassociation with the same pixel positions as in the first image.

According to another aspect of the present invention, in the imageprocessing unit, a luminance component is generated as a color componentdifferent from that of the color information in the first image.

-   (25) An image processing program according to the present invention    is used on a computer to execute an image processing step for    receiving a first image, performing weighted addition on color    information in the first image to generate a color component    different from that of the color information in the first image, and    outputting the generated color component as a second image. The    first image is expressed in a plurality of color components and    comprises a plurality of pixels each having color information    corresponding to one of the color components. In the image    processing step, at least nine coefficient patterns consisting of    values of not less than zero are prepared, and any one of the    coefficient patterns is used for the weighted addition.-   (26) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, performing weighted addition on color    information in the first image by using variable coefficients of not    less than zero to generate a color component different from that of    the color information in the first image, and outputting the    generated color component as a second image. The first-image is    expressed in a plurality of color components and comprises a    plurality of pixels each having color information corresponding to    one of the color components. In the image processing step, weighted    addition on color information present at a target pixel in the first    image and color information present at pixels adjoining the target    pixel is performed.-   (27) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, performing weighted addition on color    information in the first image by using variable coefficients of not    less than zero to generate a color component different from that of    the color information in the first image, and outputting the    generated color component as a second image. The first image is    expressed in three or more types of color components and comprises a    plurality of pixels each having color information corresponding to    one of the color components. In the image processing step, weighted    addition on color information corresponding to at least three types    of color components of the first image is performed over all the    pixels in the first image.-   (28) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, performing weighted addition on color    information in the first image by using variable coefficients of not    less than zero to generate a color component different from that of    the color information in the first image, and outputting the    generated color component as a second image. The first image is    expressed in a plurality of color components and comprises a    plurality of pixels each having color information corresponding to    one of the color components. In the image processing step, weighted    addition on the color information in the first image is performed in    constant color-component ratios over all the pixels in the first    image.-   (29) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, performing weighted addition on color    information in the first image by using variable coefficients of not    less than zero to generate a color component different from that of    the color information in the first image, performing filter    processing with predetermined fixed filter coefficients to correct    the color component different from that of the color information in    the first image, and outputting the generated color component as a    second image. The first image is expressed in a plurality of color    components and comprises a plurality of pixels each having color    information corresponding to one of the color components.-   (30) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, generating a luminance component    different from color information in the first image at the same    pixel positions as in the first image by using the color information    in the first image, generating a chrominance component different    from the color information in the first image at the same pixel    positions as in the first image separately from the luminance    component, and outputting the luminance component and the    chrominance component as a second image. The first image is    expressed in a plurality of color components and comprises a    plurality of pixels each having color information corresponding to    one of the color components. In the image processing step, weighted    addition on color information in the first image is performed by    using variable coefficients of not less than zero to generate the    luminance component.-   (31) Another image processing program according to the present    invention is used on a computer to execute an image processing step    for receiving a first image, generating a luminance component    different from color information in the first image at all of the    same pixel positions as in the first image by using the color    information in the first image, generating a chrominance component    different from the color information in the first image at the same    pixel positions as in the first image separately from the luminance    component, and outputting the luminance component and the    chrominance component as a second image. The first image is    expressed in a plurality of color components and comprises a    plurality of pixels each having color information corresponding to    one of the color components.

These and other objects and advantages of the invention may be readilyascertained by referring to the following description and appendeddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an electronic camera;

FIG. 2 illustrates the arrangement of the color components of the Bayerarrayed image data;

FIG. 3 is an operation flowchart of an image processing unit accordingto a first embodiment;

FIG. 4 is an operation flowchart of the image processing unit accordingto the first embodiment;

FIG. 5 is an operation flowchart of the image processing unit accordingto the first embodiment;

FIG. 6 is an operation flowchart of the image processing unit accordingto the first embodiment;

FIG. 7 is an illustration for explaining peripheral addition;

FIG. 8 illustrates an example of coefficient patterns;

FIG. 9 shows directions of high similarity corresponding to the valuesof (HV[i,j],DN[i,j]);

FIG. 10 illustrates an example of a coefficient pattern;

FIG. 11 illustrates an example of a band-pass filter;

FIG. 12 illustrates an example of coefficient patterns;

FIG. 13 illustrates an example of coefficient patterns;

FIG. 14 is an illustration for explaining the interpolation processingof chrominance components;

FIG. 15 is an illustration for explaining the interpolation processingof chrominance components;

FIG. 16 is a diagram showing the flow of data in the first embodiment;and

FIG. 17 is an operation flowchart of the image processing unit accordingto a second embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described indetail with reference to the drawings.

Here, a first embodiment and a second embodiment will be described inconjunction with an electronic camera having the function of imageenhancement which an image processing apparatus of the present inventionperforms.

FIG. 1 is a functional block diagram of the electronic cameracorresponding to the first embodiment and the second embodiment.

In FIG. 1, the electronic camera 1 includes an A/D conversion unit 10,an image processing unit (for example, a single-chip microprocessordedicated to image processing) 11, a control unit 12, a memory 13, acompression/decompression unit 14, and a display-image generating unit15. The electronic camera 1 also includes a memory card interface unit17 for achieving an interface with a memory card (a card type removablememory) 16, and is provided with an external interface unit 19 forachieving an interface with an external apparatus such as a PC (PersonalComputer) 18 via a specific cable or wireless transmission path. Then,these components are connected to each other via a bus.

The electronic camera 1 also includes a photographing optical system 20,an image sensor 21, an analog signal processing unit 22, and a timingcontrol unit 23. An optical image acquired by the photographing opticalsystem 20 is formed on the image sensor 21. The output of the imagesensor 21 is connected to the analog signal processing unit 22. Theoutput of the analog signal processing unit 22 is connected to the A/Dconversion unit 10. The output of the control unit 12 is connected tothe timing control unit 23. The output of the timing control unit 23 isconnected to the image sensor 21, the analog signal processing unit 22,the A/D conversion unit 10, and the image processing unit 11.

The electronic camera 1 also includes an operation unit 24 which isequivalent to a shutter release button, a selector button for modeswitching, and so on, and a monitor 25. The output of the operation unit24 is connected to the control unit 12. The output of the display-imagegeneration unit 15 is connected to the monitor 25.

The PC 18 is connected with a monitor 26, a printer 27, and so on. Anapplication program recorded in a CD-ROM 28 is previously installedtherein. Aside from its CPU, memory, and hard disk which are not shown,the PC 18 also includes a memory card interface unit (illustrationomitted) for achieving an interface with the memory card 16, and anexternal interface unit (illustration omitted) for achieving aninterface with an external apparatus such as the electronic camera 1 viaa specific cable or wireless transmission path.

In the electronic camera 1 having the configuration as in FIG. 1, when aphotographing mode is selected and the shutter release button is pressedby an operator via the operation unit 24, the control unit 12 implementstiming control over the image sensor 21, the analog signal processingunit 22, and the A/D conversion unit 10 via the timing control unit 23.The image sensor 21 generates image signals corresponding to the opticalimage. The image signals are subjected to predetermined signalprocessing in the analog signal processing unit 22, digitized by the A/Dconversion unit 10, and supplied to the image processing unit 11 asimage data.

In the electronic camera 1, R, G, and B color filters are arranged in aBayer array at the image sensor 21. The image data supplied to the imageprocessing unit 11 is thus expressed in the RGB colorimetric system inwhich each pixel contains color information corresponding to one of thethree color components, that is, a red component (corresponding to the“second color component” or “third color component” as set forth inclaims), a green component (corresponding to the “first color component”as set forth in claims), and a blue component (corresponding to the“third color component” or “second color component” as set forth inclaims). Hereinafter, such image data will be referred to as “Bayerarrayed image data”.

The image processing unit 11 receives such Bayer arrayed image data(corresponding to the “first image” as set forth in claims) andgenerates a luminance component, which is a color component of the YCbCrcolorimetric system different from the RGB colorimetric system(corresponding to the “color component different from that of the colorinformation in the first image” as set forth in claims) and chrominancecomponents at all the pixels. That is, the image processing unit 11generates a luminance plane consisting of the luminance component at allthe pixels and chrominance planes consisting of the chrominancecomponents at all the pixels, thereby achieving an enhancement of theimage. From the luminance plane and the chrominance planes thusgenerated, the image processing unit 11 also generates RGB planes ifneeded. Hereinafter, a series of processes as described above will bereferred to as an image restoration process.

It is to be noted that the image processing unit 11 performs imageprocessing including tone conversion and edge enhancement, aside fromsuch an image restoration process. After such image processing iscompleted, the image data are subjected to a specific type ofcompression processing in the compression/decompression unit 14, ifnecessary, and are recorded on the memory card 16 via the memory cardinterface unit 17.

The image data of which the image processing is completed may berecorded on the memory card 16 without compression processing, or may beconverted into the colorimetric system adopted in the monitor 26 or theprinter 27 connected to the PC 18 and may be supplied to the PC 18through the external interface unit 19.

Now, when a replay mode is selected by the operator via the operationunit 24, image data recorded on the memory card 16 are read out throughthe memory card interface unit 17, are decompressed in thecompression/decompression unit 14, and are displayed on the monitor 25via the display-image generation unit 15.

It is to be noted that the decompressed image data may be converted intothe colorimetric system adopted in the monitor 26 or the printer 27connected to the PC 18 and may be supplied to the PC 18 through theexternal interface unit 19, not being displayed on the monitor 25.

Moreover, since the processing of converting the RGB planes generated bythe image restoration process into the colorimetric system adopted inthe monitors 25, 26, and the printer 27 can be achieved by knowntechniques, description thereof will be omitted.

FIG. 2 illustrates the arrangement of the color components of the Bayerarrayed image data.

Note that, in FIG. 2, the types of the color components are representedby R, G, and B, and the positions of pixels corresponding to therespective color components by the values of the coordinates [X,Y].

Hereinafter, pixel positions with an R color filter, a G color filter,and a B color filter in the Bayer array will be referred to as an Rposition, a G position, and a B position, respectively.

Specifically, FIG. 2(1) shows the case where an R position correspondsto the processing target pixel (corresponding to the “target pixel” asset forth in claims). FIG. 2(2) shows the case where a B positioncorresponds to the processing target pixel. FIG. 2(3) shows the casewhere a G position corresponds to the processing target pixel. Among thepixels on G positions, pixels horizontally adjoining red components willbe hereinafter referred to as “pixels on Gr positions”, and pixelshorizontally adjoining blue components will be referred to as “pixels onGb positions”.

In the arithmetic expressions to be described later, when the colorinformation in each pixel is distinguished between the green componentand the other color components, R and B in FIG. 2 are replaced with Z sothat the color information in an R position or a B position is expressedas Z[i,j] and the color information in a G position is expressed asG[i,j]. Moreover, when the color information in each pixel is notdistinguished among the color components, R, G, and B in FIG. 2 isreplaced with A so that the color information in an arbitrary pixel isexpressed as A[i,j].

DESCRIPTION OF FIRST EMBODIMENT

FIGS. 3 through 6 are operation flowcharts of the image processing unit11 according to the first embodiment.

Note that FIGS. 3 through 6 show operations of the image restorationprocess out of the image processing to be performed in the imageprocessing unit 11. FIG. 3 shows an operation of the image restorationprocess in rough outline. FIGS. 4 and 5 show operations of a“luminance-component generation process” which is included in the imagerestoration process. FIG. 6 shows an operation of a“chrominance-component generation process” which is included in theimage restoration process.

Hereinafter, the first embodiment will be described. In the following,operations of the image restoration process will be explained out of theimage processing performed in the image processing unit 11. The rest ofthe operations will be omitted from the description.

Initially, in the image processing unit 11, a determination is made withregard to similarity along the vertical direction and the horizontaldirection (hereinafter, referred to as “vertical and horizontalsimilarity”) for a pixel at which the green component is missing, byusing the Bayer arrayed image data. As a result of such determination,an index HV is set for indicating the vertical and horizontal similarity(S1 in FIG. 3).

<<Example of Processing of Setting Index HV for Indicating Vertical andHorizontal similarity>>

For example, the image processing unit 11 performs the processing ofsetting the index HV for indicating the vertical and horizontalsimilarity in the following manner.

Initially, the image processing unit 11 calculates a same colorsimilarity factor Cv0 along the vertical direction and a same colorsimilarity factor Ch0 along the horizontal direction, which are definedas in Equations 1 and 2 shown below, for a pixel at which the greencomponent is missing. In addition, the image processing unit 11calculates a different color similarity factor CvN0 along the verticaldirection and a different color similarity factor ChN0 along thehorizontal direction, which are defined as in Equations 3 and 4 shownbelow, for a pixel at which the green component is missing.

<<Same Color Similarity Factors>>:Cv0[i,j]=(|G[i,j−1]−G[i,j+1]|+(|Z[i−1,j−1]−Z[i−1,j+1]|+|Z[i+1,j−1]−Z[i+1,j+1]|)/2)/2,  Eq.1Ch0[i,j]=(|G[i−1,j]−G[i+1,j]|+(|Z[i−1,j−1]−Z[i+1,j−1]|+|Z[i−1,j+1]−Z[i+1,j+1]|)/2)/2.  Eq.2<<Different Color Similarity Factors>>:CvN0[i,j]=(|G[i,j−1]−Z[i,j]|+|G[i,j+1]−Z[i,j]|)/2,  Eq. 3ChN0[i,j]=(|G[i−1,j]−Z[i,j]|+|G[i+1,j]−Z[i,j]|)/2  Eq. 4

The values calculated by using Equations 1 through 4 may be directlyused, as the same color similarity factors and the different colorsimilarity factors along the vertical and horizontal directions for thepixel on the coordinates [i,j]. Nevertheless, this example will dealwith the case where similar values are calculated not only for the pixelbut also for pixels lying around the pixel, and the individual valuesare subjected to weighted additions direction by direction (hereinafter,referred to as “peripheral addition”) to obtain the ultimate same colorsimilarity factors and different color similarity factors along thevertical direction and horizontal direction for the pixel at which thegreen component is missing.

That is, based on Equations 1 through 4, the image processing unit 11performs calculations on pixels lying on the coordinates [i,j],[i−1,j−1], [i+1,j−1], [i−1,j+1], [i+1,j+1], [i,j−2], [i,j+2], [i−2,j],and [i+2,j]. Then, the values obtained from the calculations aresubjected to peripheral addition expressed in the following Equations 5through 8, whereby a same color similarity factor Cv[i,j] along thevertical direction, a same color similarity factor Ch[i,j] along thehorizontal direction, a different color similarity factor CvN[i,j] alongthe vertical direction, and a different color similarity factor ChN[i,j]along the horizontal direction are obtained.

Note that the similarity factors obtained from Equations 5 through 8each indicate higher degree of similarity with their smaller values.Equations 5 through 8 correspond to performing peripheral addition asshown in FIG. 7.

<<Same Color Similarity Factors>>:

$\begin{matrix}{{{Cv}\left\lbrack {i,j} \right\rbrack} = \left( {{4 \cdot {{Cv0}\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{Cv0}\left\lbrack {{i - 1},{j - 1}} \right\rbrack} + {{Cv0}\left\lbrack {{i + 1},{j - 1}} \right\rbrack} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 5} \\{\left. \mspace{110mu}{{{Cv0}\left\lbrack {{i - 1},{j + 1}} \right\rbrack} + {{Cv0}\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} \right) + {{Cv0}\left\lbrack {i,{j - 2}} \right\rbrack} +} & \; \\{{\left. \mspace{110mu}{{{Cv0}\left\lbrack {i,{j + 2}} \right\rbrack} + {{Cv0}\left\lbrack {{i - 2},j} \right\rbrack} + {{Cv0}\left\lbrack {{i + 2},j} \right\rbrack}} \right)/16},} & \; \\{{{Ch}\left\lbrack {i,j} \right\rbrack} = \left( {{4 \cdot {{Ch0}\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{Ch0}\left\lbrack {{i - 1},{j - 1}} \right\rbrack} + {{Ch0}\left\lbrack {{i + 1},{j - 1}} \right\rbrack} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 6} \\{\left. \mspace{110mu}{{{Ch0}\left\lbrack {{i - 1},{j + 1}} \right\rbrack} + {{Ch0}\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} \right) + {{Ch0}\left\lbrack {i,{j - 2}} \right\rbrack} +} & \; \\{\left. \mspace{110mu}{{{Ch0}\left\lbrack {i,{j + 2}} \right\rbrack} + {{Ch0}\left\lbrack {{i - 2},j} \right\rbrack} + {{Ch0}\left\lbrack {{i + 2},j} \right\rbrack}} \right)/16.} & \;\end{matrix}$<<Different Color Similarity Factors>>:

$\begin{matrix}{{{CvN}\left\lbrack {i,j} \right\rbrack} = \left( {{4 \cdot {{CvN0}\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{CvN0}\left\lbrack {{i - 1},{j - 1}} \right\rbrack} + {{CvN0}\left\lbrack {{i + 1},{j - 1}} \right\rbrack} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 7} \\{\left. \mspace{50mu}{{{CvN0}\left\lbrack {{i - 1},{j + 1}} \right\rbrack} + {{CvN0}\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} \right) + {{CvN0}\left\lbrack {i,{j - 2}} \right\rbrack} +} & \; \\{{\left. {{{CvN0}\left\lbrack {i,{j + 2}} \right\rbrack} + {{CvN0}\left\lbrack {{i - 2},j} \right\rbrack} + {{CvN0}\left\lbrack {{i + 2},j} \right\rbrack}} \right)/16},} & \; \\{{{ChN}\left\lbrack {i,j} \right\rbrack} = \left( {{4 \cdot {{ChN0}\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{ChN0}\left\lbrack {{i - 1},{j - 1}} \right\rbrack} + {{ChN0}\left\lbrack {{i + 1},{j - 1}} \right\rbrack} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 8} \\{\left. \mspace{50mu}{{{ChN0}\left\lbrack {{i - 1},{j + 1}} \right\rbrack} + {{ChN0}\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} \right) + {{ChN0}\left\lbrack {i,{j - 2}} \right\rbrack} +} & \; \\{\left. {{{ChN0}\left\lbrack {i,{j + 2}} \right\rbrack} + {{ChN0}\left\lbrack {{i - 2},j} \right\rbrack} + {{ChN0}\left\lbrack {{i + 2},j} \right\rbrack}} \right)/16.} & \;\end{matrix}$

For example, when the pixel lying on the coordinates [i,j] is on an Rposition as shown in FIG. 2(1), the same color similarity factors Cv0and Ch0 are obtained from a G-G similarity factor component consistingof color information corresponding to the green component alone(corresponding to the “fourth similarity factor component” as set forthin claims) and a B-B similarity factor component consisting of colorinformation corresponding to the blue component alone (corresponding tothe “sixth similarity factor component” or “fifth similarity factorcomponent” as set forth in claims). In this case, the same colorsimilarity factors Cv and Ch obtained after the peripheral additioncontain an R-R similarity factor component consisting of colorinformation corresponding to the red component alone (corresponding tothe “fifth similarity factor component” or “sixth similarity factorcomponent” as set forth in claims) aside from the G-G similarity factorcomponent and the B-B similarity factor component.

Moreover, when the pixel lying on the coordinates [i,j] is on an Rposition, the different color similarity factors CvN0 and ChN0 areobtained from a G-R similarity factor component consisting of colorinformation corresponding to the green component and color informationcorresponding to the red component (corresponding to the “firstsimilarity factor component” or “third similarity factor component” asset forth in claims). In this case, the different color similarityfactors CvN and ChN obtained after the peripheral addition contain a G-Bsimilarity factor component consisting of color informationcorresponding to the green component and color information correspondingto the blue component (corresponding to the “third similarity factorcomponent” or “first similarity factor component” as set forth inclaims) aside from the G-R similarity factor component.

That is, according to the peripheral additions, similarity factors canbe obtained by taking into consideration a plurality of color componentsand also taking into consideration continuity with surrounding pixels.This means a higher degree of accuracy of the similarity factors.

By the way, in Equations 3 and 4, each term enclosed in the absolutevalue brackets is composed of color information provided at twoadjoining pixels. Thus, the different color similarity factors obtainedby Equations 3 and 4 have the function of enabling a similaritydetermination on fine structures of Nyquist frequency level whichfluctuate pixel by pixel. Moreover, these different color similarityfactors are obtained on the assumption that all the color informationcorresponding to different color components shows the same luminanceinformation. The determination on the levels of similarity using thedifferent color similarity factors is thus highly reliable on achromaticareas.

Meanwhile, the determination on the levels of similarity using the samecolor similarity factors is generally reliable on both chromatic areasand achromatic areas, whereas the reliability on areas of fine imagestructures is inferior to the case where the different color similarityfactors should be used.

Consequently, to make highly reliable similarity determination over anentire image, it is desirable that the entire image be divided intoachromatic areas and chromatic areas in which similarity factorssuitable for the respective areas are to be used.

To determine whether an image near the processing target pixel is anachromatic area or not, a color index is required which indicates thepresence or absence of local color. Among such color indexes availableis local color-difference information. The different color similarityfactors obtained as described above reflect the levels of similarity andthe local color-difference information as well. Hence, the differentcolor similarity factors can be directly used as the color index.

It should be noted that the different color similarity factors indicatehigher degree of similarity with their smaller values. Thus, when thedifferent color similarity factors along both the vertical direction andhorizontal direction have large values, they mean that the processingtarget pixel is in an achromatic area having low similarity along boththe vertical direction and horizontal direction, or the image near theprocessing target pixel is a chromatic area. On the other hand, when thedifferent color similarity factors have a relatively small value atleast along either one of the vertical direction and the horizontaldirection, it means that the image near the processing target pixel isan achromatic area and there is a direction(s) of high similarity.

The image processing unit 11 determines that the image near theprocessing target pixel is an achromatic area if a condition expressedas:CvN[i,j]≦ThNv, or ChN[i,j]≦ThNh.  Condition 1is satisfied with regard to threshold values ThNv and ThNh. If Condition1 is not satisfied, the image processing unit 11 determines that theimage near the processing target pixel is a chromatic area. Here, thethreshold values ThNv and ThNh are set to the order of 10 or lessrespectively when the gray scale is represented by 256.

When the image near the processing target pixel is an achromatic area,the image processing unit 11 then judges if a condition expressed as:|CvN[i,j]−ChN[i,j]|≦Th0.  Condition 2is satisfied or not with regard to a threshold value Th0.

Condition 2 is a condition for deciding whether or not the differentcolor similarity factor CvN[i,j] along the vertical direction and thedifferent color similarity factor ChN[i,j] along the horizontaldirection are in the same order. The threshold value Th0 has a role toavoid a noise-based misjudgment that either one of the similarities ishigh, when a difference between the different color similarity factorCvN[i,j] along the vertical direction and the different color similarityfactor ChN[i,j] along the horizontal direction is small. Consequently, afurther precise determination of similarity can be made by setting thethreshold value Th0 higher for noisy color images.

If Condition 2 is satisfied, the image processing unit 11 determinesthat the processing target pixel has low (or high) similarity along boththe vertical and horizontal directions, and sets “0” to the indexHV[i,j] for indicating the vertical and horizontal similarity. On theother hand, if Condition 2 is not satisfied, the image processing unit11 judges if the following condition expressed as:CvN[i,j]<ChN[i,j].  Condition 3is satisfied or not. If Condition 3 is satisfied, the image processingunit 11 determines that the processing target pixel has high similarityalong the vertical direction, and sets “1” to the index HV[i,j] forindicating the vertical and horizontal similarity. On the other hand, ifCondition 3 is not satisfied, the image processing unit 11 determinesthat the processing target pixel has high similarity along thehorizontal direction, and sets “−1” to the index HV[i,j] for indicatingthe vertical and horizontal similarity.

Moreover, when the image near the processing target pixel is a chromaticarea, the image processing unit 11 judges if a condition expressed as:|Cv[i,j]−Ch[i,j]|≦Th1.  Condition 4is satisfied or not with regard to a threshold value Th1.

Condition 4 is a condition for deciding whether or not the same colorsimilarity factor Cv[i,j] along the vertical direction and the samecolor similarity factor Ch[i,j] along the horizontal direction are inthe same order. The threshold value Th1 has a role to avoid anoise-based misjudgment that either one of the similarities is high,when a difference between the same color similarity factor Cv[i,j] alongthe vertical direction and the same color similarity factor Ch[i,j]along the vertical direction is small. As is the case with the thresholdvalue Th0, a further precise determination of similarity can be made bysetting the threshold value Th1 higher for noisy color images.

If Condition 4 is satisfied, the image processing unit 11 determinesthat the processing target pixel has low (or high) similarity along boththe vertical and horizontal directions, and sets “0” to the indexHV[i,j] for indicating the vertical and horizontal similarity. On theother hand, if Condition 4 is not satisfied, the image processing unit11 judges if the following condition expressed as:Cv[i,j]<Ch[i,j].  Condition 5is satisfied or not. If Condition 5 is satisfied, the image processingunit 11 determines that the processing target pixel has high similarityalong the vertical direction, and sets “1” to the index HV[i,j] forindicating the vertical and horizontal similarity. On the other hand, ifCondition 5 is not satisfied, the image processing unit 11 determinesthat the processing target pixel has high similarity along thehorizontal direction, and sets “−1” to the index HV[i,j] for indicatingthe vertical and horizontal similarity.

After the processing of setting the index HV for indicating the verticaland horizontal similarity as described above, the image processing unit11 determines the similarity along the 45° diagonal direction and 135°diagonal direction (hereinafter, referred to as “diagonal similarity”)for a pixel at which the green component is missing by using the Bayerarrayed image data. As a result of such determination, an index DN isset for indicating the diagonal similarity (S2 in FIG. 3).

<<Example of Processing of Setting Index DN for Indicating diagonalSimilarity>>

For example, the image processing unit 11 performs the processing ofsetting the index DN for indicating the diagonal similarity in thefollowing manner.

Initially, the image processing unit 11 calculates a similarity factorC45_0 along the 45° diagonal direction and a similarity factor C135_0along the 135° diagonal direction, which are defined as in Equations 9and 10 shown below, for a pixel at which the green component is missing:

$\begin{matrix}{{{C45\_}{0\left\lbrack {i,j} \right\rbrack}} = \left( \left( {{{{G\left\lbrack {i,{j - 1}} \right\rbrack} - {G\left\lbrack {{i - 1},j} \right\rbrack}}} + {{{G\left\lbrack {{i + 1},j} \right\rbrack} -}}} \right. \right.} & {{Eq}.\mspace{14mu} 9} \\{{\left. {\mspace{70mu}{G\left\lbrack {i,{j + 1}} \right\rbrack}} \right)/2} + {{{Z\left\lbrack {{i + 1},{j - 1}} \right\rbrack} - {Z\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} +} & \; \\{{\left. \mspace{70mu}{\left( {{{{Z\left\lbrack {{i + 1},{j - 1}} \right\rbrack} - {Z\left\lbrack {i,j} \right\rbrack}}} + {{{Z\left\lbrack {{i - 1},{j + 1}} \right\rbrack} - {Z\left\lbrack {i,j} \right\rbrack}}}} \right)/2} \right)/3},} & \; \\{{{C135\_}{0\left\lbrack {i,j} \right\rbrack}} = \left( \left( {{{{G\left\lbrack {i,{j - 1}} \right\rbrack} - {G\left\lbrack {{i + 1},j} \right\rbrack}}} + {{{G\left\lbrack {{i - 1},j} \right\rbrack} -}}} \right. \right.} & {{Eq}.\mspace{14mu} 10} \\{{\left. {\mspace{70mu}{G\left\lbrack {i,{j + 1}} \right\rbrack}} \right)/2} + {{{Z\left\lbrack {{i - 1},{j - 1}} \right\rbrack} - {Z\left\lbrack {{i + 1},{j + 1}} \right\rbrack}}} +} & \; \\{\left. \mspace{70mu}{\left( {{{{Z\left\lbrack {{i - 1},{j - 1}} \right\rbrack} - {Z\left\lbrack {i,j} \right\rbrack}}} + {{{Z\left\lbrack {{i + 1},{j + 1}} \right\rbrack} - {Z\left\lbrack {i,j} \right\rbrack}}}} \right)/2} \right)/3.} & \;\end{matrix}$

For example, when the pixel lying on the coordinates [i,j] is on an Rposition, the similarity factor C45 ₁₃ 0 along the 45° diagonaldirection and the similarity factor C135_0 along the 135° diagonaldirection are composed of a G-G similarity factor component, a B-Bsimilarity factor component, and an R-B similarity factor componentconsisting of the color information corresponding to the red componentand the color information corresponding to the blue component(corresponding to the “second similarity factor component” as set forthin claims) in order.

Then, the image processing unit 11 performs peripheral additions asexpressed in the following Equations 11 and 12 to obtain a similarityfactor C45 [i,j] along the 45° diagonal direction and a similarityfactor C135 [i,j] along the 135° diagonal direction. Here, Equations 11and 12 correspond to performing peripheral additions as shown in FIG. 7.

$\begin{matrix}{{{C45}\left\lbrack {i,j} \right\rbrack} = \left( {{{4 \cdot {C45\_}}{0\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{C45\_}{0\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {{C45\_}{0\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 11} \\{\left. \mspace{25mu}{{{C45\_}{0\left\lbrack {{i - 1},{j + 1}} \right\rbrack}} + {{C45\_}{0\left\lbrack {{i + 1},{j + 1}} \right\rbrack}}} \right) + {{C45\_}{0\left\lbrack {i,{j - 2}} \right\rbrack}} +} & \; \\{{\left. \mspace{25mu}{{{C45\_}{0\left\lbrack {i,{j + 2}} \right\rbrack}} + {{C45\_}{0\left\lbrack {{i - 2},j} \right\rbrack}} + {{C45\_}{0\left\lbrack {{i + 2},j} \right\rbrack}}} \right)/16},} & \; \\{{{C135}\left\lbrack {i,j} \right\rbrack} = \left( {{{4 \cdot {C135\_}}{0\left\lbrack {i,j} \right\rbrack}} + {2 \cdot \left( {{{C135\_}{0\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} +} \right.}} \right.} & {{Eq}.\mspace{14mu} 12} \\{\mspace{25mu}{{{C135\_}{0\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + \;{{C135\_}{0\left\lbrack {{i - 1},{j + 1}} \right\rbrack}} +}} & \; \\{\left. \mspace{25mu}{{C135\_}{0\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} \right) + {{C135\_}{0\left\lbrack {i,{j - 2}} \right\rbrack}} + {{C135\_}{0\left\lbrack {i,{j + 2}} \right\rbrack}} +} & \; \\{\mspace{25mu}{{C135\_}0{\left( {\left\lbrack {{i - 2},j} \right\rbrack + {{C135\_}{0\left\lbrack {{i + 2},j} \right\rbrack}}} \right)/16.}}} & \;\end{matrix}$

That is, according to the peripheral additions, the similarity factorscan be obtained with a higher degree of accuracy since a plurality ofcolor components and the continuity with surrounding pixels are takeninto account as is the case with the vertical and horizontal directions.

Next, the image processing unit 11 judges if a condition expressed as:|C45[i,j]−C135[i,j]|≦Th2.  Condition 6is satisfied or not with regard to a threshold value Th2. If Condition 6is satisfied, the image processing unit 11 determines that theprocessing target pixel has low (or high) similarity along the diagonaldirections, and sets “0” to the index DN[i,j] for indicating thediagonal similarity. On the other hand, if Condition 6 is not satisfied,the image processing unit 11 judges if the following condition expressedas:C45[i,j]<C135[i,j].  Condition 7is satisfied or not.

If Condition 7 is satisfied, the image processing unit 11 determinesthat the processing target pixel has high similarity along the 45°diagonal direction, and sets “1” to the index DN[i,j] for indicating thediagonal similarity. On the other hand, if Condition 7 is not satisfied,the image processing unit 11 determines that the processing target pixelhas high similarity along the 135° diagonal direction, and sets “−1” tothe index DN[i,j] for indicating the diagonal similarity.

After the processing of setting the index DN for indicating the diagonalsimilarity is completed as described above, the image processing unit 11performs the “luminance-component generation process” shown in FIGS. 4and 5 on each of the pixels, thereby generating a luminance plane (S3 inFIG. 3).

The “luminance-component generation process” of the first embodimentwill deal with the case where a luminance component is generated bysubjecting the color information present at a plurality of pixels lyingin a local area, out of the Bayer arrayed image data, to weightedaddition. Note that the color information to be used in such weightedaddition is determined based upon the levels of similarity at theprocessing target pixel. Coefficients for the weighted addition are setso that the luminance component contains R, G, and B in constant ratios.In the first embodiment, to achieve such a “luminance-componentgeneration process”, a plurality of types of coefficient patterns havingdifferent coefficients according to the levels of similarity(corresponding to the “at least nine coefficient patterns” as set forthin claims) are prepared.

FIG. 8 illustrates an example of the coefficient patterns used in thegeneration of a luminance component.

Here, all of α, β, u₁, u₂, v₁, v₂, s₁, s₂, t₁, and t₂ shown in FIG. 8are values of not less than zero, satisfying the following condition:α+β=1, u ₁ +u ₂=1, v ₁ +v ₂=1, s ₁ +s ₂=1, and t ₁ +t ₂=1.

The coefficient patterns shown in FIG. 8 are set so that the luminancecomponent contains R, G, and B in ratios of “β/2:α:β/2”. The colorinformation corresponding to the red component and the color informationcorresponding to the blue component are subjected to weighted additionin an identical ratio. That is, a luminance component Y satisfies therelationship:Y=α·G+β·(R+B)/2.Therefore, a common coefficient pattern can be used for pixels on Grpositions and pixels on Gb positions. Common coefficient patterns canalso be used for pixels on R positions and pixels on B positions.

Among typical examples of preferable settings of constants are asfollows:u₁≈u₂, v₁≈v₂, s₁≈s₂, and t₁≈t₂; and(α, β)=(1/3,2/3), (4/9,5/9), (5/11,6/11), (1/2,1/2), (5/9,4/9),(3/5,2/5), and (2/3,1/3).

It is to be noted that the ratios of R, G, and B constituting aluminance component are not limited to “β/2:α:β/2”, but may be “α:β:γ”(α+β+γ=1, α≧0, β≧0, γ≧0) and so on. For example, α=0.3, β=0.6, andγ=0.1. Nevertheless, when the RGB ratios are set as “α:β:γ”, coefficientpatterns must be separately provided for the pixels on Gr positions andthe pixels on Gb positions, and coefficient patterns must be separatelyprovided for the pixels on R positions and the pixels on B positions.

<<Explanation of Luminance-Component Generation Process>>

Now, with reference to FIGS. 4 and 5, the “luminance-componentgeneration process” will be explained.

Initially, the image processing unit 11 judges whether the processingtarget pixel is on a G position or not (S11 in FIG. 4).

When the processing target pixel is not on a G position (NO at S11 inFIG. 4), the image processing unit 11 judges what values the indexHV[i,j] for indicating the vertical and horizontal similarity and theindex DN[i,j] for indicating the diagonal similarity at the processingtarget pixel have (S12 in FIG. 4), and classifies the levels ofsimilarity at the processing target pixel into any one of case 1 to case9 shown below:

case 1: (HV[i,j],DN[i,j])=(0,0): high similarity along all thedirections, or low similarity along all the directions (the direction ofhigh similarity is undetermined);

case 2: (HV[i,j],DN[i,j])=(0,1): high similarity along the 45° diagonaldirection;

case 3: (HV[i,j],DN[i,j])=(0,−1): high similarity along the 135°diagonal direction;

case 4: (HV[i,j],DN[i,j])=(1,0): high similarity along the verticaldirection;

case 5: (HV[i,j],DN[i,j])=(1,1): high similarity along the vertical and45° diagonal directions;

case 6: (HV[i,j],DN[i,j])=(1,−1): high similarity along the vertical and135° diagonal directions;

case 7: (HV[i,j],DN[i,j])=(−1,0): high similarity along the horizontaldirection;

case 8: (HV[i,j],DN[i,j])=(−1,1): high similarity along the horizontaland 45° diagonal directions; and

case 9: (HV[i,j],DN[i,j])=(−1,−1): high similarity along the horizontaland 135° diagonal directions.

FIG. 9 shows directions of high similarity corresponding to the valuesof (HV[i,j],DN[i,j]).

According to the determined results mentioned above, the imageprocessing unit 11 generates a luminance component as follows:

When the processing target pixel is classified as case 1, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 1 (S13 in FIG. 4);

When the processing target pixel is classified as case 2, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 2 (S14 in FIG. 4);

When the processing target pixel is classified as case 3, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 3 (S15 in FIG. 4);

When the processing target pixel is classified as case 4, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 4 (S16 in FIG. 4);

When the processing target pixel is classified as case 5, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 5 (S17 in FIG. 4);

When the processing target pixel is classified as case 6, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 6 (S18 in FIG. 4);

When the processing target pixel is classified as case 7, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 7 (S19 in FIG. 4);

When the processing target pixel is classified as case 8, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 8 (S20 in FIG. 4); and

When the processing target pixel is classified as case 9, the imageprocessing unit 11 generates a luminance component by using thecoefficient pattern 9 (S21 in FIG. 4).

That is, the luminance component Y[i,j] is generated through theoperation of any one of the following Equations 13 through 21, using acoefficient pattern selected from the nine coefficient patterns 1through 9 in accordance with the cases 1 through 9:

Coefficient pattern 1:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 13} \\{\mspace{101mu}{\alpha \cdot \left( {{\left( {v_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {\left( {v_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}} +} \right.}} & \; \\{\left. \mspace{101mu}{{\left( {u_{1}/2} \right) \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {\left( {u_{2}/2} \right) \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right) +} & \; \\{\mspace{101mu}{\left( {\beta/2} \right) \cdot \left( {{\left( {s_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {\left( {s_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} +} \right.}} & \; \\{\left. \mspace{101mu}{{\left( {t_{1}/2} \right) \cdot {A\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + {\left( {t_{2}/2} \right) \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right);} & \;\end{matrix}$

Coefficient pattern 2:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 14} \\{\mspace{101mu}{\alpha \cdot \left( {{\left( {v_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {\left( {v_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}} +} \right.}} & \; \\{\left. \mspace{101mu}{{\left( {u_{1}/2} \right) \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {\left( {u_{2}/2} \right) \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right) +} & \; \\{\mspace{101mu}{{\left( {\beta/2} \right) \cdot \left( {{t_{1} \cdot {A\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + {t_{2} \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right)};}} & \;\end{matrix}$

Coefficient pattern 3:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 15} \\{\mspace{101mu}{\alpha \cdot \left( {{\left( {v_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {\left( {v_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}} +} \right.}} & \; \\{\left. \mspace{101mu}{{\left( {u_{1}/2} \right) \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {\left( {u_{2}/2} \right) \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right) +} & \; \\{\mspace{101mu}{{\left( {\beta/2} \right) \cdot \left( {{s_{1} \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {s_{2} \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}}} \right)};}} & \;\end{matrix}$

Coefficient pattern 4:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 16} \\{\mspace{101mu}{{\alpha \cdot \left( {{u_{1} \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {u_{2} \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right)} + {\left( {\beta/2} \right) \cdot}}} & \; \\{\mspace{101mu}\left( {{\left( {s_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {\left( {s_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} +} \right.} & \; \\{\left. \mspace{101mu}{{\left( {t_{1}/2} \right) \cdot {A\left\lbrack {{i + 1^{1}},{j - 1}} \right\rbrack}} + {\left( {t_{2}/2} \right) \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right);} & \;\end{matrix}$

Coefficient pattern 5:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 17} \\{\mspace{101mu}{{\alpha \cdot \left( {{u_{1} \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {u_{2} \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right)} +}} & \; \\{\mspace{95mu}{{\left( {\beta/2} \right) \cdot \left( {{t_{1} \cdot {A\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + {t_{2} \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right)};}} & \;\end{matrix}$

Coefficient pattern 6:

$\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} & {{Eq}.\mspace{14mu} 18} \\{\mspace{101mu}{{\alpha \cdot \left( {{u_{1} \cdot {A\left\lbrack {i,{j - 1}} \right\rbrack}} + {u_{2} \cdot {A\left\lbrack {i,{j + 1}} \right\rbrack}}} \right)} +}} & \; \\{\mspace{95mu}{{\left( {\beta/2} \right) \cdot \left( {{s_{1} \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {s_{2} \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}}} \right)};}} & \;\end{matrix}$

Coefficient pattern 7:

$\begin{matrix}\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} \\{{\alpha \cdot \left( {{v_{1} \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {v_{2} \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}}} \right)} +} \\{\left( {\beta/2} \right) \cdot \left( {{\left( {s_{1}/2} \right) \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {\left( {s_{2}/2} \right) \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}} +} \right.} \\{\left. {{\left( {t_{1}/2} \right) \cdot {A\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + {\left( {t_{2}/2} \right) \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right);}\end{matrix} & {{Eq}.\mspace{14mu} 19}\end{matrix}$

Coefficient pattern 8:

$\begin{matrix}\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} \\{{\alpha \cdot \left( {{v_{1} \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {v_{2} \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}}} \right)} +} \\{{\left( {\beta/2} \right) \cdot \left( {{t_{1} \cdot {A\left\lbrack {{i + 1},{j - 1}} \right\rbrack}} + {t_{2} \cdot {A\left\lbrack {{i - 1},{j + 1}} \right\rbrack}}} \right)};}\end{matrix} & {{Eq}.\mspace{14mu} 20}\end{matrix}$

Coefficient pattern 9:

$\begin{matrix}\begin{matrix}{{Y\left\lbrack {i,j} \right\rbrack} = {{\left( {\beta/2} \right) \cdot {A\left\lbrack {i,j} \right\rbrack}} +}} \\{{\alpha \cdot \left( {{v_{1} \cdot {A\left\lbrack {{i - 1},j} \right\rbrack}} + {v_{2} \cdot {A\left\lbrack {{i + 1},j} \right\rbrack}}} \right)} +} \\{\left( {\beta/2} \right) \cdot {\left( {{s_{1} \cdot {A\left\lbrack {{i - 1},{j - 1}} \right\rbrack}} + {s_{2} \cdot {A\left\lbrack {{i + 1},{j + 1}} \right\rbrack}}} \right).}}\end{matrix} & {{Eq}.\mspace{14mu} 21}\end{matrix}$

In each of the coefficient patterns described above, the colorinformation present at the processing target pixel and the colorinformation present at pixels adjoining the processing target pixel(including the pixels adjoining along the diagonal directions) can beadded with weights. The position of the color information to besubjected to the weighted addition and the values of the coefficientsare changed in accordance with the differences in similarity. Fromanother point of view, the weighted addition can be performed on thecolor information in a narrowest range of pixels which contain all thecolor information corresponding to R, G, and B, three color componentsby using the coefficients.

Now, when the processing target pixel is on a G position, the luminancecomponent at the processing target pixel can be generated by using acoefficient pattern 10 shown in FIG. 8 (corresponding to the“coefficient pattern different from the at least nine coefficientpatterns” as set forth in claims) irrespective of the levels ofsimilarity such as described above. Nevertheless, when the processingtarget pixel is on a G position, and both the index HV[i−1,j] forindicating the vertical and horizontal similarity and the indexDN[i−1,j] for indicating the diagonal similarity at the pixel adjoiningthe left of the processing target pixel are “0”, the generation of theluminance component using the coefficient pattern 10 can sometimes causea false structure of unpleasant checkered pattern which does not existin reality. Then, the first embodiment provides an example of performingweighted addition on the color information corresponding to the greencomponent at the processing target pixel and the color informationcorresponding to a green component lying the closest to the processingtarget pixel, and applying a low-pass filter to the green componentselectively in generating the luminance component so that the occurrenceof such a false structure is suppressed.

Specifically, when the image processing unit 11 ascertains from thejudgement of S11 in FIG. 4 that the processing target pixel is on a Gposition, it judges whether or not the index HV[i−1,j] for indicatingthe vertical and horizontal similarity and the index DN[i−1,j] forindicating the diagonal similarity at the pixel adjoining the left ofthe processing target pixel are both “0” (S22 in FIG. 5).

When the index HV[i−1,j] for indicating the vertical and horizontalsimilarity and the index DN[i−1,j] for indicating the diagonalsimilarity are both “0”, the image processing unit 11 then calculatesthe following Equation 22 to generate a luminance component Y[i,j] byusing a coefficient pattern 0 of FIG. 10 in which a low-pass filter isapplied to the green component (corresponding to the “coefficientpattern for performing weighted addition on the color informationpresent at a plurality of pixels having the first color component” asset forth in claims) (S23 in FIG. 5):Y[i,j]=α·(d ₁ ·G[i−1,j−1]+d ₂ ·G[i,j]+d ₃ ·G[i−1,j+1])+(β/4)·Z[i−1,j]+Z[i+1,j]+Z[i,j−1]+Z[i,j+1])  Eq. 22

Here, in Equation 22, α, β, d₁, d₂, and d₃ are values of not less thanzero. The intensity of the low-pass filter can be set by changing thevalues of d₁, d₂, and d₃ while satisfying the relationship d₁+d₂+d₃=1.For example, among possible values of d₁, d₂, and d₃ for applying astrong low-pass filter are “1, 6, and 1”. Among possible values of d₁,d₂, and d₃ for applying a weak low-pass filter are “1, 14, and 1”.

On the other hand, when at least either one of the index HV[i−1,j] forindicating the vertical and horizontal similarity and the indexDN[i−1,j] for indicating the diagonal similarity is not “0” (NO at S22in FIG. 5), the image processing unit 11 generates a luminance componentby using the coefficient pattern 10 (S24 in FIG. 5). That is, theluminance component Y[i,j] is generated through the operation of thefollowing Equation 23:Y[i,j]=α·A[i,j]+(β/4)·(A[i−1,j]+A[i+1,j]+A[i,j−1]+A[i,j+1])  Eq. 23

Having performed the “luminance-component generation process” asdescribed above to generate a luminance plane, the image processing unit11 performs an edge enhancement process (corresponding to the “filterprocessing with predetermined fixed filter coefficients” as set forth inclaims) on the luminance plane to correct the luminance plane (S4 inFIG. 3).

<<Example of Processing for Correcting the Luminance Planes>>

For example, the image processing unit 11 performs the edge enhancementprocess by adding the results of band-pass filtering to the luminanceplane by use of a filter containing coefficients of positive andnegative values as shown in FIG. 11, to the original luminance plane.That is, the correction of the luminance plane is realized through theoperations of the following Equations 24 and 25:YH[i,j]=(8·Y[i,j]−(Y[i−1,j]+Y[i+1,j]+Y[i,j−1]+Y[i,j+1]+Y[i−1,j−1]+Y[i+1,j−1]+Y[i−1,j+1]+Y[i+1,j+1]))/16.  Eq.24Y[i,j]=Y[i,j]+K·YH[i,j].  Eq. 25

Here, in Equation 25, K is a coefficient for changing the intensity ofthe edge enhancement and is assumed to be a value of the order of 1.

Incidentally, the luminance plane generated thus can be output as amonochrome image.

Next, the image processing unit 11 performs the “chrominance-componentgeneration process” shown in FIG. 6 on the pixels on R positions andpixels on B positions, thereby generating a Cr component of the pixelson the R positions and a Cb component of the pixels on the B positions(S5 in FIG. 3).

In the “chrominance-component generation process” of the firstembodiment, a chrominance component is generated by subjecting the colorinformation at a plurality of pixels lying in a local area, out of theBayer arrayed image data, to weighted addition directly. Incidentally,the color information to be used in such weighted addition and thecoefficients of the weighted addition are determined based upon thelevels of similarity at the processing target pixel. In the firstembodiment, coefficient patterns are provided in advance for performingsuch a “chrominance-component generation process”.

FIGS. 12 and 13 illustrate an example of the coefficient patterns to beused in the chrominance component generation.

Incidentally, as compared to the coefficient patterns shown in FIG. 12,the coefficient patterns shown in FIG. 13 are more effective insuppressing the occurrence of color artifacts.

<<Explanation of Chrominance-Component Generation Process>>

Now, with reference to FIG. 6, the “chrominance-component generationprocess” will be explained.

Initially, the image processing unit 11 judges what value the indexHV[i,j] for indicating the vertical and horizontal similarity at theprocessing target pixel (pixel on an R position or pixel on a Bposition) has (S31 in FIG. 6). Then, in accordance with the value of theindex HV[i,j] for indicating the vertical and horizontal similarity, theimage processing unit 11 generates the chrominance component in thefollowing manner:

For “HV[i,j]=1”, the image processing unit 11 generates a chrominancecomponent by using a coefficient pattern 11 (S32 in FIG. 6);

-   -   For “HV[i,j]=−1”, the image processing unit 11 generates a        chrominance component by using a coefficient pattern 12 (S33 in        FIG. 6); and    -   For “HV[i,j]=0”, the image processing unit 11 generates a        chrominance component by using a coefficient pattern 13 (S34 in        FIG. 6).

More specifically, when the coefficient patterns shown in FIG. 12 areused, the Cr component Cr[i,j] of the processing target pixel on an Rposition is generated through the operation of the following Equations26 through 28, depending on the value of the HV[i,j]. Moreover, when thecoefficient patterns shown in FIG. 13 are used, the Cr component Cr[i,j]of the processing target pixel on an R position is generated through theoperation of the following Equations 29 through 31, depending on thevalue of the HV[i,j].

<<In the Case Where Coefficient Patterns Shown in FIG. 12 are Used>>:HV[i,j]=1:Cr[i,j]=Z[i,j]−(G[i,j−1]+G[i,j+1])/2;  Eq. 26HV[i,j]=−1:Cr[i,j]=Z[i,j]−(G[i−1,j]+G[i+1,j])/2;  Eq. 27HV[i,j]=0:Cr[i,j]=Z[i,j]−(G[i,j−1]+G[i,j+1]+G[i−1,j]+G[i+1,j])/4.  Eq.28<<In the Case Where Coefficient Patterns Shown in FIG. 13 are Used>>:HV[i,j]=1:Cr[i,j]=(2·Z[i,j]+Z[i,j−2]+Z[i,j+2])/4−(G[i,j−1]+G[i,j+1])/2;  Eq.29HV[i,j]=−1:Cr[i,j]=(2·Z[i,j]+Z[i−2,j]+Z[i+2,j])/4−(G[i−1,j]+G[i+1,j])/2;  Eq.30HV[i,j]=0:Cr[i,j]=(4·Z[i,j]+Z[i,j−2]+Z[i,j+2]+Z[i−2,j]+Z[i+2,j])/8−(G[i,j−1]+G[i,j+1]+G[i−1,j]+G[i+1,j])/4.  Eq.31

Moreover, when the processing target pixel is on a B position, the Cbcomponent Cb[i,j] is generated through Equations 26 to 31 as well.

Having performed the “chrominance-component generation process” asdescribed above over all the pixels on R positions and all the pixels onB positions, the image processing unit 11 interpolates the Cr componentand the Cb component for pixels at which the Cr component or Cbcomponent is missing, thereby generating chrominance planes (S6 in FIG.3). Here, the processing for interpolating the Cr component will beexplained while the processing for interpolating the Cb component willbe omitted, since the processing for interpolating the Cr component andthe processing for interpolating the Cb component can be performed inthe same manner.

<<Example of Processing for Interpolating the Cr Component>>

For example, the image processing unit 11 calculates the Cr componentfor pixels at which the Cr component is missing by averaging the Crcomponent at a plurality of pixels on R positions lying around thepixel, thereby realizing the interpolation of the Cr component.Specifically, for a pixel on a B position, an interpolation value of theCr component is obtained through Equation 32 shown below. Moreover, fora pixel on a Gr position, an interpolation value of the Cr component isobtained through Equation 33 shown below. For a pixel on a Gb position,an interpolation value of the Cr component is obtained through Equation34 shown below.

<<Pixel on B Position>>:Cr[i,j]=(Cr[i−1,j−1]+Cr[i−1,j+1]+Cr[i+1,j−1]+Cr[i+1,j+1])/4;  Eq. 32<<Pixel on Gr Position>>:Cr[i,j]=(Cr[i−1,j]+Cr[i+1,j])/2;  Eq. 33<<Pixel on Gb Position>>:Cr[i,j]=(Cr[i,j−1]+Cr[i,j+1])/2.  Eq. 34

By the way, color artifacts attributable to color moire that occurs froma periodic structure of an image can be reduced by applying a low-passfilter along with the interpolating calculation of the Cr componentvalues.

For example, such processing can be achieved by calculating the Crcomponent for a pixel on an R position through Equation 35 seen below,calculating the Cr component for a pixel on a B position throughEquation 36 shown below, calculating an interpolation value of the Crcomponent for a pixel on a Gr position through Equation 37 shown below,and calculating an interpolation value of the Cr component for a pixelon a Gb position through Equation 38 shown below.

<<Pixel on R Position>>:

$\begin{matrix}\begin{matrix}{{{Cr}\left\lbrack {i,j} \right\rbrack} = \left( {{36 \cdot {{Cr}\left\lbrack {i,j} \right\rbrack}} +} \right.} \\{6 \cdot \left( {{{Cr}\left\lbrack {{i - 2},j} \right\rbrack} + {{Cr}\left\lbrack {{i + 2},j} \right\rbrack} + {{Cr}\left\lbrack {i,{j - 2}} \right\rbrack} +} \right.} \\{\left. {{Cr}\left\lbrack {i,{j + 2}} \right\rbrack} \right) + {1 \cdot \left( {{{Cr}\left\lbrack {{i - 2},{j - 2}} \right\rbrack} + {{Cr}\left\lbrack {{i + 2},{j - 2}} \right\rbrack} +} \right.}} \\{{\left. \left. {{{Cr}\left\lbrack {{i - 2},{j + 2}} \right\rbrack} + {{Cr}\left\lbrack {{i + 2},{j + 2}} \right\rbrack}} \right) \right)/64};}\end{matrix} & {{Eq}.\mspace{14mu} 35}\end{matrix}$<<Pixel on B Position>>:Cr[i,j]=(16·(Cr[i−1,j−1]+Cr[i+1,j−1]+Cr[i−1,j+1]+Cr[i+1,j+1]))/64;  Eq.36<<Pixel on Gr Position>>:Cr[i,j]=(24·(Cr[i−1,j]+Cr[i+1,j])+4·(Cr[i−1,j−2]+Cr[i+1,j−2]+Cr[i−1,j+2]+Cr[i+1,j+2]))/64;  Eq. 37<<Pixel on Gb position>>:Cr[i,j]=(24·(Cr[i,j−1]+Cr[i,j+1])+4·(Cr[i−2,j−1]+Cr[i+2,j−1]+Cr[i−2,j+1]+Cr[i+2,j+1]))/64.  Eq. 38

It is to be noted that if the Cr component at the pixels on G positionsand the Cr component at the pixels on B positions are initialized to“0”, the operation of Equations 32 through 34 can be achieved byapplying such a filter as shown in FIG. 14, and the operation ofEquations 35 through 38 can be achieved by applying such a filter asshown in FIG. 15.

In the manner described above, the Cr component and Cb component areinterpolated to generate the chrominance planes, which can be output asa YCbCr color image. If necessary, the image processing unit 11 performsa colorimetric system conversion process on the luminance plane and thechrominance planes to generate RGB planes (S7 in FIG. 3).

<<Example of Processing for Generating RGB Planes>>

Through the colorimetric system conversion process, the image processingunit 11 achieves the generation of RGB planes, for example, byperforming the operation of Equations 39 through 41 shown below:R[i,j]=Y[i,j]+(1−β/2)·Cr[i,j]−(β/2)·Cb[i,j],  Eq. 39G[i,j]=Y[i,j]−(β/2)·Cr[i,j]−(β/2)·Cb[i,j],  Eq. 40B[i,j]=Y[i,j]−(β/2)·Cr[i,j]+(1−β/2)·Cb[i,j].  Eq. 41

FIG. 16 is a diagram showing the flow of data according to the firstembodiment.

In the first embodiment, the luminance component at pixels on Rpositions and that at pixels on B positions are generated by usingcoefficient patterns that are selected from among the nine coefficientpatterns depending on the similarity. It is therefore possible togenerate an extremely smooth luminance plane even on fine edges.

Moreover, in the first embodiment, weighted addition can be achievedusing the color information corresponding to the R, G, and B, threecolor components at the processing target pixel and pixels adjoining theprocessing target pixel in a narrowest range, taking into considerationthe similarity along various directions. This gives excellentsuitability for the “luminance-component generation process” on Bayerarrayed image data, allowing the generation of a smooth luminance plane.

Besides, in the first embodiment, coefficient patterns that are set sothat a luminance component contains R, G, and B in constant ratios areused through the “luminance-component generation process”. Consequently,it is possible to prevent a nonexistent structure in the photographicobject (a structure attributable to the Bayer array) from appearing as astructure that varies pixel by pixel. It is therefore possible togenerate a luminance plane with higher precision than the conventionalart in which the ratios of R, G, and B constituting a luminance valuehave been uneven pixel by pixel.

Moreover, in the first embodiment, the luminance plane that carries finestructures is subjected to the edge enhancement process to correct thehigh frequency components. It is therefore possible to obtain aluminance plane with higher resolution than the conventional art inwhich the correction of high frequency components has been insufficient.

Furthermore, in the first embodiment, since the chrominance planes aregenerated directly from the Bayer arrayed image data independently ofthe luminance plane, they will not be affected by the frequencycharacteristics of the luminance plane, which might cause the occurrenceof color artifacts, unlike in the conventional art where the“chrominance values” have been generated with reference to luminancevalues. It is therefore possible to generate the chrominance planes withhigh precision.

Besides, as is evident from FIG. 16, in the first embodiment, theindexes for indicating similarity are obtained directly from the Bayerarrayed image data. The first embodiment thus differs from theconventional art where the directions of similarity at individual pixelshave been classified with reference to “blurred luminance values” whosehigh frequency components as to structural factors are all broken. Inthe first embodiment, the similarity is determined by using the Bayerarrayed image data directly. It is therefore possible to generate aluminance plane with a high resolution in areas of fine structures.

As has been described, according to the first embodiment, imageenhancement is achieved with high precision.

Moreover, in the first embodiment, the similarity is taken into accountin generating the luminance plane. The correction of high frequencycomponents can thus be achieved by a band pass filter with fixedcoefficients. This eliminates the need for the correction usingdifferent correction values in accordance with the directions ofsimilarity at respective pixels, which has been exercised in theconventional art, and can reduce the reference planes used in thecorrection processing. Consequently, when the generation of theluminance plane is realized by hardware, the hardware structure can bemore simplified than in the conventional art.

Incidentally, in the first embodiment, the Cr component and the Cbcomponent are interpolated to generate the chrominance planes at S6 inFIG. 3. Such chrominance planes may be subjected to correctionprocessing for reducing the occurrence of color artifacts further (suchas processing of applying a median filter).

DESCRIPTION OF SECOND EMBODIMENT

Hereinafter, description will be given of an operation of the secondembodiment.

Differences between the second embodiment and the first embodimentconsist in that the second embodiment does not require the setting ofthe index DN for indicating diagonal similarity (the processingcorresponding to S2 in FIG. 3) which has been exercised in the firstembodiment, and that the “luminance-component generation process” isdifferent.

Then, in the second embodiment, description will be given of the“luminance-component generation process”. Description of the rest of theoperation will be omitted.

FIG. 17 is an operation flowchart of the image processing unit I Iaccording to the second embodiment, showing the operation of the“luminance-component generation process” in particular.

<<Explanation of Luminance-Component Generation Process>>

Hereinafter, the “luminance-component generation process” will beexplained with reference to FIG. 17.

Note that, in the “luminance-component generation process” of the secondembodiment, a luminance component is generated by subjecting the colorinformation at a plurality of pixels lying in a local area, out of theBayer arrayed image data, to weighted addition as in the firstembodiment. To perform such a “luminance-component generation process”,the coefficient patterns 1, 4, 7, and 10 out of the coefficient patternsshown in FIG. 8 and the coefficient pattern 0 shown in FIG. 9 areprepared in advance.

Initially, as in the first embodiment, the image processing unit 11judges whether or not the processing target pixel is on a G position(S41 in FIG. 17). When the processing target pixel is not on a Gposition (NO at S41 in FIG. 17), the image processing unit 11 judgeswhat value the index HV[i,j] for indicating the vertical and horizontalsimilarity of the processing target pixel has (S42 in FIG. 17). Then,depending on the value of the index HV[i,j] for indicating the verticaland horizontal similarity, the image processing unit 11 generates aluminance component in the following manner.

For “HV[i,j]=0”, the image processing unit 11 generates a luminancecomponent by using the coefficient pattern 1 (S43 in FIG. 17). That is,the luminance component Y[i,j] is generated through the operation ofEquation 13 in the first embodiment.

For “HV[i,j]=1”, the image processing unit 11 generates a luminancecomponent by using the coefficient pattern 4 (S44 in FIG. 17). That is,the luminance component Y[i,j] is generated through the operation ofEquation 16 in the first embodiment.

For “HV[i,j]=−1”, the image processing unit 11 generates a luminancecomponent by using the coefficient pattern 7 (S45 in FIG. 17). That is,the luminance component Y[i,j] is generated through the operation ofEquation 19 in the first embodiment.

Now, when the processing target pixel is on a G position, the imageprocessing unit 11 judges whether or not the index HV[i−1,j] forindicating the vertical and horizontal similarity at the pixel adjoiningthe left of the processing target pixel is “0” (S46 in FIG. 17).

Then, when the index HV[i−1,j] for indicating the vertical andhorizontal similarity is “0”, the image processing unit 11 generates aluminance component Y[i,j] by using the coefficient pattern 0 as in thefirst embodiment (S47 in FIG. 17). That is, the luminance componentY[i,j] is generated through the operation of Equation 22 in the firstembodiment.

On the other hand, when the index HV[i−1,j] for indicating the verticaland horizontal similarity is not “0”, the image processing unit 11generates a luminance component by using the coefficient pattern 10 asin the first embodiment (S48 in FIG. 17). That is, the luminancecomponent Y[i,j] is generated through the operation of Equation 23 inthe first embodiment.

As has been described, the second embodiment omits the setting of theindex DN for indicating diagonal similarity which has been exercised inthe first embodiment. The number of coefficient patterns is thus reducedfrom in the first embodiment.

Consequently, according to the second embodiment, an extremely smoothluminance plane even on fine edges along various directions, thoughsomewhat inferior to the first embodiment, can be generated through aprocedure simpler than in the first embodiment, thereby achieving imageenhancement with high precision.

Incidentally, in the luminance-component generation processes of thefirst and second embodiments, different coefficient patterns areprovided for applying a low-pass filter selectively in accordance withthe directional similarity at the pixel adjoining a processing targetpixel on the left, when the processing target pixel is on a G position.In an ordinary image restoration process alone, however, a luminancecomponent of the processing target pixel on a G position may begenerated by using the coefficient pattern 1 alone irrespective of suchdirectional similarity. That is, in the first embodiment, the processingof S24 in FIG. 5 may be exclusively performed instead of the processingof S22 through S24 in FIG. 5. Moreover, in the second embodiment, theprocessing of S48 in FIG. 17 may be exclusively performed instead of theprocessing of S46 through S48 in FIG. 17.

THIRD EMBODIMENT

Hereinafter, description will be given of an operation of the thirdembodiment.

In the third embodiment, image processing is performed on the PC 18shown in FIG. 1.

Here, an image processing program recorded on the CD-ROM 28 or otherrecording medium (an image processing program for performing an imagerestoration process as the image processing unit 11 in any of theforegoing embodiments does) is installed on the PC 18 in advance. Thatis, a not-shown hard disk in the PC 18 contains such an image processingprogram as executable on a not-shown CPU.

Hereinafter, the operation of the third embodiment will be describedwith reference to FIG. 1.

Initially, when a photographing mode is selected and a shutter releasebutton is pressed by an operator through the operation unit 24, theelectronic camera 1 digitizes image signals that are generated in theimage sensor 21 and given predetermined analog signal processing in theanalog signal processing unit 22, with the A/D conversion unit 10 andsupplies them to the image processing unit 11 as image data. The imageprocessing unit 11 performs such image processing as tone conversion andgamma correction on the image data supplied thus. The image data ofwhich such image processing is completed are recorded on the memory card16 via the memory card interface unit 17.

Next, in a state where a PC communication mode is selected by theoperator through the operation unit 24, the transfer of image data isinstructed from the PC 18 via the external interface unit 19. Then, theelectronic camera 1 reads the image data corresponding to theinstruction from the memory card 16 through the memory card interfaceunit 17. Then, the image data read thus is supplied to the PC 18 throughthe external interface unit 19.

The not-shown CPU in the PC 18, when supplied with the image data inthis way, executes the image processing program mentioned above.Incidentally, image data given the image restoration process through theexecution of such image processing program may be recorded on thenot-shown hard disk, or may be converted into a colorimetric systemadopted in the monitor 26 or the printer 27 if needed.

As has been described, according to the third embodiment, the same imagerestoration process as in any of the foregoing embodiments can beperformed by the PC 18, so that image enhancement is achieved with highprecision.

Incidentally, when a memory card 16 having image data recorded thereonas described previously is loaded, the not-shown CPU in the PC 18 mayread the image data from the memory card 16 and execute the foregoingimage processing program.

Moreover, such an image processing program may be downloaded to the PC18 through access to a predetermined Web site on the Internet.

Besides, such an image processing program may be executed by a remoteserver or the like connected to the Internet or the like, not by the PC18. That is, the PC 18 has only to transfer the image data supplied fromthe electronic camera 1, to a server or the like capable of executingthe foregoing image processing program over the Internet or the like.Then, the image data can be subjected to the same image restorationprocessing as in any of the foregoing embodiments.

Each of the foregoing embodiments has dealt with the processing on animage that is expressed in the RGB colorimetric system and each pixelthereof contains the color information corresponding to any single colorcomponent among R, G, and B. The same processing can also be applied,however, to images that are expressed in other colorimetric systems.

Furthermore, each of the foregoing embodiments has dealt with the casewhere the processing is targeted on the image data that has colorcomponents patterned as shown in FIG. 2. Nevertheless, the patterns ofimage data to which the present invention is applicable are not limitedto those shown in FIG. 2.

INDUSTRIAL APPLICABILITY

According to the image processing apparatus of the present invention, itis possible to achieve a high-precision enhancement of an image that isexpressed in a plurality of color components and consists of a pluralityof pixels each having color information corresponding to one of thecolor components.

In addition, according to the image processing program of the presentinvention, it is possible to achieve, by using a computer, ahigh-precision enhancement of an image that is expressed in a pluralityof color components and consists of a plurality of pixels each havingcolor information corresponding to one of the color components.

1. An image processing apparatus comprising an image processing unit forreceiving a first image, performing weighted addition of colorinformation in said first image to generate a color component differentfrom that of the color information in said first image, and outputtingthe generated color component as a second image, said first image beingexpressed in a plurality of color components and comprising a pluralityof pixels each having color information corresponding to one of thecolor components in a specific manner, wherein: in said image processingunit, at least nine coefficient patterns consisting of values of notless than zero are prepared, and any one of said coefficient patterns isused for the weighted addition.
 2. The image processing apparatusaccording to claim 1, wherein: in said image processing unit, levels ofsimilarity along a plurality of directions are determined, and which touse from said at least nine coefficient patterns is selected inaccordance with the determined result.
 3. The image processing apparatusaccording to claim 1, wherein: in said image processing unit, at leastnine coefficient patterns are prepared for performing weighted additionon color information present at a target pixel in said first image andcolor information present at pixels adjoining said target pixel.
 4. Theimage processing apparatus according to claim 1, wherein: in said imageprocessing unit, when said first image is expressed in a first colorcomponent set with a higher pixel density and second and third colorcomponents set with a lower pixel density, weighted addition on a pixelhaving said first color component is performed by using a coefficientpattern prepared separately from said at least nine coefficientpatterns.
 5. The image processing apparatus according to claim 4,wherein: in said image processing unit, levels of similarity along aplurality of directions are determined, for a pixel having said secondor third color component and adjacent to pixels having said first colorcomponent, and when the levels of similarity are indistinguishable alongany direction, a coefficient pattern including weighted addition oncolor information present at a plurality of pixels having the firstcolor component is used as said coefficient pattern prepared separately.6. The image processing apparatus according to claim 5, wherein: in saidimage processing unit, at the time of the weighted addition, acoefficient pattern including weighted addition of color informationpresent at a target pixel and color information present at the pixelshaving said first color component and lying the closest to said targetpixel is used, as said coefficient pattern including the weightedaddition on the color information present at the plurality of pixelshaving the first color component.
 7. An image processing apparatuscomprising an image processing unit for receiving a first image,performing weighted addition of color information in said first image byusing variable coefficients of not less than zero to generate a colorcomponent different from that of the color information in said firstimage, and outputting the generated color component as a second image,said first image being expressed in a plurality of color components andcomprising a plurality of pixels each having color informationcorresponding to one of the color components in a specific manner,wherein: in said image processing unit, weighted addition of colorinformation present at a target pixel in said first image and colorinformation present at pixels adjoining said target pixel is performed.8. The image processing apparatus according to claim 7, wherein: in saidimage processing unit, levels of similarity along a plurality ofdirections are determined, and said coefficients of the weightedaddition are changed in accordance with the determined result.
 9. Animage processing apparatus comprising an image processing unit forreceiving a first image, performing weighted addition of colorinformation in said first image by using variable coefficients of notless than zero to generate a color component different from that of thecolor information in said first image, and outputting the generatedcolor component as a second image, said first image being expressed inthree or more types of color components and comprising a plurality ofpixels each having color information corresponding to one of the colorcomponents in a specific manner, wherein: in said image processing unit,weighted addition of color information corresponding to at least threetypes of color components in said first image is performed over all thepixels in said first image.
 10. The image processing apparatus accordingto claim 9, wherein: in said image processing unit, levels of similarityalong a plurality of directions are determined, and said coefficients ofthe weighted addition are changed in accordance with the determinedresult.
 11. The image processing apparatus according to claim 9,wherein: in said image processing unit, weighted addition on colorinformation present at pixels in a narrowest range around a target pixelin said first image is performed, said narrowest range including thecolor information corresponding to said at least three types of colorcomponents.
 12. An image processing apparatus comprising an imageprocessing unit for receiving a first image, performing weightedaddition of color information in said first image by using variablecoefficients of not less than zero to generate a color componentdifferent from that of the color information in said first image, andoutputting the generated color component as a second image, said firstimage being expressed in a plurality of color components and comprisinga plurality of pixels each having color information corresponding to oneof the color components in a specific manner, wherein: in said imageprocessing unit, weighted addition of the color information in saidfirst image in constant color-component ratios is performed over all thepixels in said first image.
 13. The image processing apparatus accordingto claim 12, wherein: in said image processing unit, levels ofsimilarity along a plurality of directions are determined, and saidcoefficients of the weighted addition are changed in accordance with thedetermined result.
 14. The image processing apparatus according to claim12, wherein: in said image processing unit, when said first image isexpressed in a first color component set with a higher pixel density andsecond and third color components set with a lower pixel density,weighted addition of color information corresponding to said secondcolor component and color information corresponding to said third colorcomponent is performed at an identical color-component ratio.
 15. Animage processing apparatus comprising an image processing unit forreceiving a first image, performing weighted addition of colorinformation in said first image by using variable coefficients of notless than zero to generate a color component different from that of thecolor information in said first image, performing filter processing withpredetermined fixed filter coefficients to correct said color componentdifferent from that of the color information in said first image, andoutputting the generated color component as a second image, said firstimage being expressed in a plurality of color components and comprisinga plurality of pixels each having color information corresponding to oneof the color components in a specific manner.
 16. The image processingapparatus according to claim 15, wherein: in said image processing unit,levels of similarity along a plurality of directions are determined, andsaid coefficients of the weighted addition are changed in accordancewith the determined result.
 17. The image processing apparatus accordingto claim 15, wherein: in said image processing unit, filter coefficientsincluding positive and negative values are used as said predeterminedfixed filter coefficients.
 18. An image processing apparatus comprisingan image processing unit for receiving a first image, generating aluminance component different from color information in said first imageat the same pixel positions as in the first image by using the colorinformation in said first image, generating a chrominance componentdifferent from the color information in said first image at the samepixel positions as in the first image separately from said luminancecomponent, and outputting said luminance component and said chrominancecomponent as a second image, said first image being expressed in aplurality of color components and comprising a plurality of pixels eachhaving color information corresponding to one of the color components ina specific manner, wherein: in said image processing unit, weightedaddition of color information in said first image is performed by usingvariable coefficients of not less than zero to generate said luminancecomponent.
 19. An image processing apparatus comprising an imageprocessing unit for receiving a first image, generating a luminancecomponent different from color information in said first image at all ofthe same pixel positions as in the first image by using the colorinformation in said first image, generating a chrominance componentdifferent from the color information in said first image at the samepixel positions as in the first image separately from said luminancecomponent, and outputting said luminance component and said chrominancecomponent as a second image, said first image being expressed in aplurality of color components and comprising a plurality of pixels eachhaving color information corresponding to one of the color components ina specific manner.
 20. The image processing apparatus according to claim18, wherein: in said image processing unit, levels of similarity along aplurality of directions are determined, and said luminance component isgenerated by performing weighted addition of the color information insaid first image in accordance with the determined result.
 21. The imageprocessing apparatus according to claim 18, wherein: in said imageprocessing unit, levels of similarity along a plurality of directionsare determined, and said chrominance component is generated byperforming weighted addition of the color information in said firstimage in accordance with the determined result.
 22. The image processingapparatus according to claim 20, wherein: in said image processing unit,when said first image is expressed in a first color component, a secondcolor component, and a third color component, similarity factors along aplurality of directions are calculated by using at least one similarityfactor component out of: a first similarity factor component consistingof said first color component and said second color component; a secondsimilarity factor component consisting of said second color componentand said third color component; a third similarity factor componentconsisting of said third color component and said first color component;a fourth similarity factor component consisting of said first colorcomponent alone; a fifth similarity factor component consisting of saidsecond color component alone; and a sixth similarity factor componentconsisting of said third color component alone, and the levels ofsimilarity along said plurality of directions are determined based onsaid similarity factors.
 23. The image processing apparatus according toclaim 1, wherein: in said image processing unit, said color component ofsaid second image is outputted in association with the same pixelpositions as in said first image.
 24. The image processing apparatusaccording to claim 1, wherein: in said image processing unit, aluminance component is generated as a color component different fromthat of the color information in said first image.
 25. A computerreadable medium storing an image processing program for outputting acolor component for enhancing an image, the image processing programcomprising instructions which, when executed on a computer, cause thecomputer to execute: an image processing step for receiving a firstimage, a step of performing weighted addition of color information insaid first image to generate a color component different from that ofthe color information in said first image, and a step of outputting thegenerated color component as a second image, said first image beingexpressed in a plurality of color components and comprising a pluralityof pixels each having color information corresponding to one of thecolor components in a specific manner, wherein: in said image processingstep, at least nine coefficient patterns consisting of values of notless than zero are prepared, and any one of said coefficient patterns isused for the weighted addition.
 26. A computer readable medium storingan image processing program for outputting a color component forenhancing an image, the image processing program comprising instructionswhich, when executed on a computer, cause the computer to execute: animage processing step for receiving a first image, a step of performingweighted addition of color information in said first image by usingvariable coefficients of not less than zero to generate a colorcomponent different from that of the color information in said firstimage, and a step of outputting the generated color component as asecond image, said first image being expressed in a plurality of colorcomponents and comprising a plurality of pixels each having colorinformation corresponding to one of the color components in a specificmanner, wherein: in said image processing step, weighted addition ofcolor information present at a target pixel in said first image andcolor information present at pixels adjoining said target pixel isperformed.
 27. A computer readable medium storing an image processingprogram for outputting a color component for enhancing an image, theimage processing program comprising instructions which, when executed ona computer, cause the computer to execute: an image processing step forreceiving a first image, a step of performing weighted addition of colorinformation in said first image by using variable coefficients of notless than zero to generate a color component different from that of thecolor information in said first image, and a step of outputting thegenerated color component as a second image, said first image beingexpressed in three or more types of color components and comprising aplurality of pixels each having color information corresponding to oneof the color components in a specific manner, wherein: in said imageprocessing step, weighted addition of color information corresponding toat least three types of color components of said first image isperformed over all the pixels in said first image.
 28. A computerreadable medium storing an image processing program for outputting acolor component for enhancing an image, the image processing programcomprising instructions which, when executed on a computer, cause thecomputer to execute: an image processing step for receiving a firstimage, a step of performing weighted addition of color information insaid first image by using variable coefficients of not less than zero togenerate a color component different from that of the color informationin said first image, and a step of outputting the generated colorcomponent as a second image, said first image being expressed in aplurality of color components and comprising a plurality of pixels eachhaving color information corresponding to one of the color components ina specific manner, wherein: in said image processing step, weightedaddition of the color information in said first image is performed inconstant color-component ratios over all the pixels in said first image.29. A computer readable medium storing an image processing program foroutputting a color component for enhancing an image, the imageprocessing program comprising instructions which, when executed on acomputer, cause the computer to execute: an image processing step forreceiving a first image, a step of performing weighted addition of colorinformation in said first image by using variable coefficients of notless than zero to generate a color component different from that of thecolor information in said first image, a step of performing filterprocessing with predetermined fixed filter coefficients to correct saidcolor component different from that of the color information in saidfirst image, and a step of outputting the generated color component as asecond image, said first image being expressed in a plurality of colorcomponents and comprising a plurality of pixels each having colorinformation corresponding to one of the color components in a specificmanner.
 30. A computer readable medium storing an image processingprogram for outputting a luminance component and a chrominance componentfor enhancing an image, the image processing program comprisinginstructions which, when executed on a computer, cause the computer toexecute: an image processing step for receiving a first image, a step ofgenerating a luminance component different from color information insaid first image at the same pixel positions as in the first image byusing the color information in said first image, a step of generating achrominance component different from the color information in said firstimage at the same pixel positions as in the first image separately fromsaid luminance component, and a step of outputting said luminancecomponent and said chrominance component as a second image, said firstimage being expressed in a plurality of color components and comprisinga plurality of pixels each having color information corresponding to oneof the color components in a specific manner, wherein: in said imageprocessing step, weighted addition of color information in said firstimage is performed by using variable coefficients of not less than zeroto generate said luminance component.
 31. A computer readable mediumstoring an image processing program for outputting a luminance componentand a chrominance component for enhancing an image, the image processingprogram comprising instructions which, when executed on a computer,cause the computer to execute: an image processing step for receiving afirst image, a step of generating a luminance component different fromcolor information in said first image at all of the same pixel positionsas in the first image by using the color information in said firstimage, a step of generating a chrominance component different from thecolor information in said first image at the same pixel positions as inthe first image separately from said luminance component, and a step ofoutputting said luminance component and said chrominance component as asecond image, said first image being expressed in a plurality of colorcomponents and comprising a plurality of pixels each having colorinformation corresponding to one of the color components in a specificmanner.
 32. The image processing apparatus according to claim 7,wherein: in said image processing unit, said color component of saidsecond image is outputted in association with the same pixel positionsas in said first image.
 33. The image processing apparatus according toclaim 9, wherein: in said image processing unit, said color component ofsaid second image is outputted in association with the same pixelpositions as in said first image.
 34. The image processing apparatusaccording to claim 12, wherein: in said image processing unit, saidcolor component of said second image is outputted in association withthe same pixel positions as in said first image.
 35. The imageprocessing apparatus according to claim 15, wherein: in said imageprocessing unit, said color component of said second image is outputtedin association with the same pixel positions as in said first image. 36.The image processing apparatus according to claim 19, wherein: in saidimage processing unit, levels of similarity along a plurality ofdirections are determined, and said luminance component is generated byperforming weighted addition of the color information in said firstimage in accordance with the determined result.
 37. The image processingapparatus according to claim 36, wherein: in said image processing unit,when said first image is expressed in a first color component, a secondcolor component, and a third color component, similarity factors along aplurality of directions are calculated by using at least one similarityfactor component out of: a first similarity factor component consistingof said first color component and said second color component; a secondsimilarity factor component consisting of said second color componentand said third color component; a third similarity factor componentconsisting of said third color component and said first color component;a fourth similarity factor component consisting of said first colorcomponent alone; a fifth similarity factor component consisting of saidsecond color component alone; and a sixth similarity factor componentconsisting of said third color component alone, and the levels ofsimilarity along said plurality of directions are determined based onsaid similarity factors.
 38. The image processing apparatus according toclaim 19, wherein: in said image processing unit, levels of similarityalong a plurality of directions are determined, and said chrominancecomponent is generated by performing weighted addition of the colorinformation in said first image in accordance with the determinedresult.